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Abstract 

In this paper we focus on the convergence analysis of the forward-backward splitting method for 
solving nonsmooth optimization problems in Hilbert spaces when the objective function is the 
sum of two convex functions. Assuming that one of the functions is Frechet differentiable and 
using two new linesearches, the weak convergence is established without any Lipschitz continuity 
assumption on the gradient. Furthermore, we obtain many complexity results of cost values at 
the iterates when the stepsizes are bounded below by a positive constant. 
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1 Introduction 

We are interested in solving problems of the following form: 

min /(x) -b g{x) subject to x £71, (1) 

where 7i is a real Hilbert space with the inner product (•, •), and f,g:7i^ IR := IR U {+oo} 
are two proper lower semicontinuous convex functions in which / is Frechet differentiable on an 
open set containing the domain of g. The optimal solution set of this problem will be denoted 
by 5*. Recently problem ([T]) together with many variants of it has received much attention from 
optimization community due to its broad applications to many disciplines such as optimal control, 
signal processing, system identification, machine learning, and image analysis; see, e.g., [igiisiiST] 
and the references therein. Many effective methods have been proposed to solve problem ([1]). Most 
of them keep using the idea of splitting / and g separately and taking the advantage of some 
Lipschitz assumption on the derivative of / at each iteration. Here we focus our attention on the 
so-called forward-backward splitting method, which contains a forward gradient step of / (an explicit 
step) followed by a backward proximal step of g (an implicit step) for problem ([T]); see, e.g., [27] . 
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In this work linesearches are used to eliminate the undesired Lipschitz assumption on the gradient 
of / mostly imposed in the literature. 

To describe and motivate our methods, let us recall here the so-called proximal operator prox^ := 
{dg + Id)“^, where dg is the classical convex subdifferential of g and Id is the identity operator in 
H. Among many important properties of proximal operators, it is well-known that prox^ is well- 
defined with full domain, single-valued, and even nonexpansive; see, e.g., [3l ll5l[T6] . Furthermore, 
for any a > 0, x is an optimal solution to problem ([T]) if and only if x = prox„g(x — aV/(x)). This 
indeed motivates the construction of the iterative sequence forming the forward-backward iteration 
as following: 

:= prox„^g(x^ - afcV/(x^)) (2) 

with positive stepsize Uk- The iteration presented in ([2]) has been attracted extensive interests due to 
its simplicity and several important advantages. It is well-known that this method uses little storage, 
readily exploits the separable structure of problem ([T|) , and is easily implemented to practical appli¬ 
cations; see [5 | l27 p 29|. Moreover, scheme ([2]) may reduce to many popular optimization methods as 
particular cases including the projected gradient method for smooth constrained minimization; the 
proximal point method; the CQ algorithm for the split feasibility problem; the projected Landweber 
algorithm for constrained least squares; the iterative soft thresholding algorithm for linear inverse 
problems; decomposition methods for solving variational inequalities; and the simultaneous orthog¬ 
onal projection algorithm for the convex feasibility problem; see, e.g., [ll[T3l[T71[T8l[32l[35l|36] and 
the references therein. 

The convergence of the iteration ([2]) to an optimal solution of ([I|) is usually established under the 
assumption that the gradient of / is Lipschitz continuous and the stepsize is taken bounded below 
and less than some constant related with the Lipschitz modulus; see, e.g., [m Theorem 3.4(i)]. In 
this case, the main machinery to prove the convergence and its complexity is based on the renowned 
Baillon-Haddad Theorem [31 Corollary 18.16]. When V/ is Lipschitz continuous but somehow the 
Lipschitz constant is not known, finding the stepsize ak that guarantees the convergence of ([2]) would 
be a challenge. However, the following linesearch proposed in |5] overcome this inconvenience: 
choosing the stepsize ak in ([2]) as the largest a £ {u, <70, (T0^, ..., } with constants a > 0 and 
6 G (0,1) such that: 

/(J(x^a)) < /(x^) + (V/(x^), J(x^a) - x^) + ^||x^ - J(x^a)f, (3) 

where J(x^, a) := proxQ,g(x^ — aV/(x^)) and || • || is the norm induced by the inner product in H. 
This linesearch is well-defined by taking the advantage of the Lipschitz assumption for V/ again via 
the so-called Descent Lemma [31 Theorem 18.15(iii)]. As far as we observe, the theory of convergence 
and complexity for the forward-backward is almost complete under such a Lipschitz assumption. 
However, the Lipschitz condition fails in many natural circumstances; see, e.g., [H]. It is quite 
interesting to question the convergence of the method and its complexity without the Lipschitz 
assumption aforementioned. In [33] Tseng provided an evidence of positive answer even for more 
general problems of finding a zero point of the sum of two maximal monotone operators. His crucial 
approach motivates us to construct Method [T] for problem ([T]) in our Section 4. But working on the 
functionals (/ and g) rather than just the maximal operators (V/ and dg) actually gives us much 
more convenience. Indeed, we completely relax an (expensive) extra projection step from Tseng’s 
scheme and omit several unnatural assumptions in the main theorem [341 Theorem 3.4]. Moreover, 
in the spirit of linesearch on functionals like ([3]) and following some ideas presented in [71 I331I37] , we 
also introduce a new linesearch mainly used in our Method [3] in Section 5. Both Method [T]and 
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Method [3] guarantee weak convergence of their generated sequences to optimal solutions without 
imposing the Lipschitz assumption on V/. 

Another achievement of our work is the study on complexity of cost values at generated sequences, 
which are proved to converge to the infimum value of problem ([T|) even in the case when the set of 
optimal solutions is empty. It is worth mentioning that in order to obtain the rate 0{k~^) of the 
functional value {f + g){x^) to the optimal cost, the gradient V/ is usually supposed to be globally 
Lipschitz continuous in the classical forward-backward iteration [iEKISlETlEg]. Here, in finite 
dimensions, we derive the better rate o{k~^) even with strictly weaker assumptions, for instance, 
V/ only needs to be locally Lipschitz continuous for our Method [T] and Method [3l This partially 
generalizes several results in [2IH23], in which the authors also derive the complexity o{k for 
proximal point method (when / = 0). Moreover, we present an interesting example of problem ([T]) 
with non-Lipschitz gradient where the stepsizes generated by both linesearches converge to zero 
and the complexity o{k~^) of the cost values remains valid. Furthermore, the rate O {k~‘^'j is also 
obtained for our Method [2l an accelerating version of Method [1] motivated from [5]. Again, 
global Lipschitz continuity on V/ is lessened. 

The paper is organized as follows. The next section presents some preliminary results that will 
be used throughout the paper. We also discuss here our standing assumptions for the problem 
which is somewhat natural for the lack of Lipschitz assumption aforementioned. Section [3] devotes 
to the two different linesearches for the forward-backward methods used in Sections 0] and [5l Weak 
convergence and complexity of the forward-backward method with the first linesearch are analyzed 
in Section m We also consider its accelerated version here. Section [5] provides a similar study for 
a variant of the forward-backward splitting method with the second linesearch. We complete the 
paper with some conclusion for further study. 


2 Preliminary results 


In this section we present some definitions and results needed for our paper. Let h : T-L ^ Wl 
be a proper, lower semicontinuous (Z.s.c.), and convex function. We denote the domain of h by 
dom/i := {x ^ Ti \ h{x) < -|-oo}. For any x G dom/i, the directional derivative of /i at x in the 
direction d is 


h'(x]d) := lim 
t^-o+ 


h{x + td) — h{x) 


which always exists (although it may be infinite). The subdifferential of /i at x is defined by 


dh{x) := {v € Hi {v,y — x) < h{y) — h{x), y G %}. 

Fact 2.1 ( O Proposition 17.2] ). Let h : LL ^ ]R, he a proper, l.s.c., and convex function, 
for X G dom/i and y gH, the following hold: 


(i) h'{x]y) exists and h'{x‘,y) = inf 


h{x + ty) — h{x) 


(4) 

Then, 


(ii) h'{x\ y - x) + h{x) < h{y). 

Fact 2.2 ( dni Theorem 4.7.1 and Proposition 4.2.1(i)] ). The subdifferential operator dh is maxi¬ 
mal monotone, i.e., it has no proper monotone extension in the graph inclusion sense. Moreover, 
the graph of dh, Gph((9/i) := {(x,x) G x 7/| x G dh{x)} is demiclosed, i.e., if the sequence 
(x^,x^)fcgN C Gph(cl/i) satisfies that (x^)fcgN converges weakly to x and (x^)fcgN converges strongly 
to V, then (x,x) G Gph(cl/i). 


3 




Next we set the standing assumptions on the data of problem ([T]) used throughout the paper as 
follows: 

A1 /,5 : —>• ]R are two proper l.s.c. convex functions with dom^ C dom/. 

A2 The function / is Frechet differentiable on an open set containing dom 5 . The gradient V/ is 
uniformly continuous on any bounded subset of dom s' and maps any bounded subset of domg' 
to a bounded set in %. 

Assumption ! All and the first part of Assumption ! A2l are popular and crucial for the well-definedness 
of the forward-backward iteration ([2]). It is easy to check that the second part of !A2! is automatic 
when V/ is Lipschitz continuous on dom 5 :. However, Assumption A2 is not enough to guarantee 
the Lipschitz continuity of V/. Indeed, the convex functions f{x) = ||x||p (1 < p < -|-oo, p / 2) 
and g{x) = 0, x £ H satisfy all the conditions in !A2! but V/ is not globally Lipschitz continuous. 
When Ti is a finite-dimensional space and the domain of g is closed. Assumption ! A2! actually means 
that / is Frechet differentiable on an open set containing dom 5 ' and that its gradient is continuous 
on dom g. It is worth noting further that the closedness of dom g is broadly assumed for problem 
© in the literature including the case of optimization problems with geometric constraints, which 
can be written as when g is an indicator function; see, e.g., m- 

Proposition 2.3. Let LL he a finite-dimensional space and let f,g : LL ^ Wl be two functions 
satisfying !A 1 [ Suppose that the closure of dom g, denoted by cl (dom is a subset of dom f, f is 
Frechet differentiable on an open set containing cl (dom 5 ), and that its gradient ^ f is continuous 
on cl (dom ( 7 ). Then Assumption ! A2! is satisfied. 

Consequently, ifdomg is closed then the validity of Assumption \A^ is equivalent to the statement 
that f is Frechet differentiable on an open set containing dom 5 : and its gradient is continuous on 
domg. 

Proof. To justify, suppose that dim'H < -|-oo, cl (domg) C A C dom/, / is Frechet differentiable 
on an open set containing cl (domg), and that V/ is continuous on cl (domg). Take any bounded 
set A of domg. Note that V/ is uniformly continuous on the compact set clA C cl (domg) and 
thus on A due to the classical Ffeine-Cantor Theorem. Since V/ is continuous on cl (domg), it 
maps the compact set cl A C cl (dom g) to a compact set in Ti. This verihes that V/(A) is bounded 
and completes the first part of the proposition. 

Now suppose that dom g is closed. It is easy to see that the validity of Assumption !A2! implies 
that V/ is continuous on domg. This together with the first part of this proposition justifies the 
second part. The proof is completed. □ 

Let us recall the proximal operator prox^ : FL —domg with proXg( 2 :) = (Id-|-clg)”^( 2 :), z £ FL. 
It is well-known that the proximal operator is single-valued with full domain. Furthermore, note 
that 

z — prox„„(2;) 

- - - £ (9g(prox (z)) for all z £FL, a £ 1R++ := {t G ]R| t > 0}. (5) 

a ^ 

We also denote the forward-backward operator J : domg x 1R++ —)• domg C by 

J{x,a) := prox„g(x — aV/(x)) for all x £ domg C dom/, a > 0. ( 6 ) 

The following lemma is very useful for our further study. 
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Lemma 2.4 ( |24l Lemma 1]). Let /, 5 ^ : "H —?• IR 6 e two functions satisfying Assumvtion \ All Then 
for any x G dom^r and 02 > ai > 0, we have 

Q!2 

— ||x — J(x,ai)|| > \\x — J{x^a 2 )\\ > ||x — J(x,ai)||. ( 7 ) 

Ol\ 

Let us end the section by recalling the well-known concepts of so-called quasi-Fejer and Fejer 
convergence. The definition originates in m and has been elaborated further in |12[l25j . 

Definition 2.1. Let S be a nonempty subset ofTi. A sequence {x^)k&^ in TL is said to be quasi- 
Fejer convergent to S if and only if for all x £ S there exists a sequence {ek)keN in 1R+ such that 
YlT=o < +00 and — x|p < \\x^ — x|p -|- for all A: G N. When {ek)k£N is a null sequence, 

we say that {x^)k£f^ is Fejer convergent to S. 

Fact 2.5 ( [25l Theorem 4.1] ). If (x^)fcGN is quasi-Fejer convergent to S, then one has: 

(i) The sequence {x^)keN is bounded. 

(ii) If all weak accumulation points of {x^)keN belong to S, then {x^)k&i is weakly convergent to a 
point in S. 

3 The linesearches 

In this section we present two different linesearches mainly used in the forward-backward methods 
proposed in Sections 01 and O The first one contains a backtracking procedure which computes at 
least one backward step (implicit step) inside the updating inner loop for finding the steplength. 
This linesearch is a particular case of the one proposed in |34] for solving inclusion problems. It 
will be used in Method [Tjand Method [2] of Section 0] 


Linesearch 1. Given x, a > ^, 9 £ (0,1) and 6 £ (0,1/2). 

Input. Set a = a and J{x,a) := prox^g{x — aVf{x)) with x £ domg. 

While q; II V/( J(a:, a)) — V/(a;) || > 5 \\J{x,a) — x\\ do 
a = 9a. 

End While 
Output, a. 

The well-definedness of Linesearch 0] follows from m Theorem 3.4(a)]. For the reader’s conve¬ 
nience, we provide a different proof revealing that the convexity of / is not necessary. 

Lemma 3.1. If x £ dom^ then Linesearch 0] stops after finitely many steps. 


Proof. If X G S* then x = J{x,a). Thus the linesearch stops with zero step and gives us the output 
a. If X ^ S'*, by contradiction suppose that for all a G P := {a, a9, a9 '^,...}, 


a ||V/(j(x, a)) — V/(x)|| > (5 II J(x, Of) — x|| . (8) 

When a £V is sufficiently closed to 0, it follows from Lemma [2.4l that J(x, a) is uniformly bounded. 
Thus we get from ([8|) that ||x — J(x, a) || —)• 0 as a | 0 thanks to Assumption I A21 The latter implies 
||V/(j(x,a)) — V/(x)|| —>■ 0 when a | 0 by Assumption I A2I again. Thus we get from ([8|) that 


lim 

otO 


X — J(x, a)|| 
a 


= 0 . 


(9) 
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Employing ([5]) with z = x — aSJf{x) gives us that 

X—J(x,a) „ / T/ NN 

- - e Vf{x) + dg{J{x,a)). 

a 

By letting a | 0 in the above inclusion and using ([9]), we get from the demiclosedness of Gph(9(7) 
from Fact 12.21 that 0 £ V/(x) + dg{x) C d{f + g){x). This contradicts the assumption that x is 
not an optimal solution to problem ([T]) and completes the proof of the lemma. □ 

Next we propose the second backtracking procedure. In contrast to Linesearch 1, this linesearch 
demands only one evaluation of the backward step and uses it in all possible iterations. This is 
somehow an advantage of this linesearch, since in many practical problems computing the proximal 
operator many times may be very expensive. The linesearch is indeed a generalization of the one 
studied in [7] for solving the nonlinear constrained optimization problem [g = 5c)- We will employ 
it in Method [3] in Section [5j 


Linesearch 2. Given x and 6 G (0,1). 

Input. Set (3 = 1, Jx '■= J{x, 1) = proXg(x — V/(x)) with x G dom^f. 

While {f + g) (x - /3(x - Jx)) > (f + g)(x) - /3 lg(x) - g(Jx)j - /3{Vf{x),x- Jx) + ^\\x - Jx 

do 

p = e(3. 

End While 
Output. (3. 


Similarly to Linesearch [H we also have finite termination for Linesearch [2j It is important to 
note that the well-definedness analysis is done without assuming the second part of IA2I (uniform 
continuity and boundedness). 

Lemma 3.2. If x £ domg then Linesearch [2] stops after finitely many steps. 


Proof. If x G 5* we have x = Jx- Thus the linesearch immediately gives us the output 1 without 
proceeding any step. If x ^ S'*, by contradiction let us assume that Linesearch [2] does not stop 
after finitely many steps. Thus for all f3 G Q := {1,0,0^,...}, we have 


{f + g){x - f3{x - Jx)) > if + g){x) - 13 [g{x) - g{Jx)] - /3(V/(x),x - Jx) + ^11^ - Jxf- 


It follows that 

if + 9){x- (3{x - Jx)) - (/ + g){x) 


1 , 


+ gix) - g{Jx) + (V/(x),x - Jx) > -|k - Jo 


^ ■ y. . v- . , - - u,, - 

Taking (3 \.D and using the Frechet differentiability of / and the convexity of g give us that 


1 , 


\x - Jx\? <(v/(x), Jx-x) + g'{x\ Jx - x) + g{x) - g{Jx) + (V/(x),x - Jx) 
=g'{x] Jx-x) + g{x) - g{Jx) < 0, 


where the last inequality follows from Fact I2.11 iii. Hence we have x = Jx, which readily implies 
that X —V/(x) G dg{x) + x, i.e., 0 G Vf{x) + dg{x) C d{f + g){x). This contradicts the assumption 
X 0 S*. □ 
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4 The forward-backward method with Linesearch [T] 

This section devotes to the study of the forward-backward splitting method with Linesearch [TJ We 
mainly derive the weak convergence of the generated sequences from this method and also obtain 
the same complexity of [5l Theorem 1.1] for the cost value sequences generated from the forward- 
backward iteration under a weaker assumption than the Lipschitz one on V/ usually imposed in 
the literature. 

The following method has some similarities to the one proposed in |34j for maximal monotone 
operators. However, it completely relaxes an extra expensive projection step [Ml Equation (2.3)] 
and seems to be more natural in comparison with the classical forward-backward splitting method 

(Ell- 


Method 1. 

Initialization Step. Take G dom^i, u > 0, 0 G (0,1) and 5 G (0,1/2). 

Iterative Step. Given set 

= J(x^, Ofc) ;= prox„^^(a:^ - afcV/(x^)) (10) 

with ak '■= Linesearch [T](x^, cr, 9, 5). 

Stop Criteria. If x^~^^ = x’^, then stop. 


First note that from Lemma (3. II that Linesearch [T] for finding the stepsize ak in the above scheme 
is finite. Hence the choice of sequence {x^)kefi hr Method [1] is well-defined. Another important 
feature from the definition of Linesearch [T] useful for our analysis is the following inequality 




V/(x"+i) - Vf{x^) 


< <5 




— X 


( 11 ) 


Note further that if Method [T] stops at iteration k then we have x^ = ]irox.^^g{x^ — Uk'^f{x^)) 
and consequently G 5*. Otherwise, we will mainly show that the sequence {x^)k&^ generated 
by this method is converging weakly to some optimal solution. Verifying this claim needs some 
auxiliary results as follows. 

Proposition 4.1. Let ak = Linesearch [T](x^, <7,0,5). For all k and x G domgi, we have 

(i) IJx^ — x]]^ — llx^+^ — x]]^ > 2ak [(/ -I- 5)(x^+^) — {f + fl')(a;)] + (1 — 25)jjx^+^ — x^]]^; 

(ii) (/ + -{f + g){x^) < - - x^JJ^. 


Proof. First let us justify (i) by noting from ([5]) and (fTOl) that 

.fc - J(x^ 

Oik 


_ ^k-\-l 


O^k 


_ v/(a;^) = _ v/(x'=) G dg{J{x^,ak)) = dg{x^+^). 


It follows from the convexity of g that 

I h 

X — X 


j k k I 1 \ 

g{x) — g{x^~^^) > / -V/(x^), X — \ for all x&domg. 


( 12 ) 
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Since / is convex, we also have 


f{x)-f{y)>{'^f{y),x-y) for all xGdomf, y£ doing. (13) 

Summing (I12p and (jl3p with any x G dom g C dom/ and y = x^ (z dom g gives us the following 
expressions 


(/ + ff)(x) >f{x^) + g{x^^^) + (-V/(x"), X - ) + (V/(x"), X - x") 

\ / 

=/(x^) + g{x^^^) + —(x^ - x^+\ X - x^+^) + (V/(x^), x^+^ - x^) 
ak 

>f{x^) + gix’"^^) + —(x^ - x^+\ X - x^+^) + (V/(x^+i), x'=+^ - x^) 

ak 

-||V/(x'=)-V/(x'^+i)||-||x'^+i-x'^|| 

>/(a;^) + 9{x^^^) + —{x^- x^+\ X - x^+^) + (V/(x^+^), x^+^ - x^) 
ak 

ak 


where the last inequality follows from dill) . After rearrangement we get 

(x^-x^+Sx^+^-x) > ak[f{x^)+g{x’^+^)-{f+g){x) + {Vf{x^+^),x^+^-x'^)]-6\\x’^+^-x^f. (14) 

Since 2(x^ — x^+^, x^+^ — x) = ||x^ — x|p — ||x^'''^ — x|p — ||x^ — x^+^|p, we get from (fHp that 

||x^ - x|p - ||x^+^ - x|p > 2Q;fc[/(x^)+ 5 (x^+^) - (/+ 5 )(x)] . . 

+2afc(V/(x'=+^),x'=+^-x'=) + (l-2,5)||x^-x^+if. ^ ’ 

By using (fTBp with x = x^ and y = we have /(x^) — /(x^+^) > (V/(x^'''^),x^ — x^+^). This 

together with (IT^ gives us that 

_ a;||2 _ _ ^||2 >2ak[{f + 5 )(x"+') - (/ + 9 ){x)] + (1 - 2,5)||x'^ - x^+^f, 

which verihes (i). Note further that (ii) is a consequence of (i) when x = x^. The proof is 
complete. □ 


Proposition Id.ll iii shows that Method [T] is a descent method in the sense that the value of 
the cost function / + at each iteration is decreasing. Furthermore, it is easy to check from 
Proposition I4.11 ii that the generated sequence of Method [1] is Fejer convergent to the optimal 
solution set S* whenever S* 7 ^ 0. This observation is indeed the center of the following main result 
of this section, where we prove the weak convergence of sequence (x^)fcgpj in Method [T]and also 
{{f+ 9)ix^))km is a minimizing sequence of / + 5 ^ without the Lipschitz assumption on V/. To 
the best of our knowledge, this result improves [5l Theorem 1.2] and even the classical results for 
gradient method with linesearch; see, for instance, [U Proposition 1.3.3] and [1]. Moreover, we show 
that the sequence ((/ + g){x '^))converges to the infimum value when the solution set is empty. 

Theorem 4.2. Let (x^)fcgN o,nd {ak)kGn be the sequences generated by Method [TJ The following 
statements hold: 

(i) If ^ 9 then (x^)fegi^ is weakly convergent to a point in S^. Moreover, 

lim (/ + g){x'^) = min (/ + g){x). 
k—^oo x£H 


(16) 



(ii) If = ij) then we have 

lim ||x^|| =+00 and lim {f + g){x^) = inf {f + g){x). 

k—^oo k^oo xGl-L 


Proof. Let us justify (i) by supposing that 5* 7 ^ 0. By applying Proposition I4.1lf i) at any x* £ S'*, 
we have 


^ > “laklif + g){x^^^) - [f + 9){x*)] + (1 - 25)||x^ - (17) 

> (1 - 25)||x'' - x^^^f > 0. 


It follows that the sequence {x^)k£n is FejCT convergent to S* and thus is bounded by Fact I2.5f ii. 
By using (fTTD . we get 

0 < 2ak[{f + g){x^^^) - {f + g){x^)] < \\x^ - x*|p - - x*|p 

= (||x^ — x*|| + ||x^+^ — x*||) • (||x^ — x*|| — llx^’*'^ — x*||) 

< 2M(||x^ - x*|| - llx^’*'^ - x*||) 

< 2M||x^ -x^+^ll, 


where M := sup{||x^ “ 3:^*lll A: £ N} < + 00 . Hence the above inequalities lead us to 

U + 9 ){x^"^^) - {f + g){x^) < M— - - -(18) 

Oik 

Due to the Fejer property of {x^)keN to <5*, the sequence (||a;^—x*||)fcgN is convergent. This together 
with (fT7)l tells us that ||x^ — x^+^H —^ 0 as fc — 00 . 

Since (x^)^^^ is bounded, the set of its weak accumulation points is nonempty. Take any weak 
accumulation point x of {x^)k£n, we find a subsequence (x”'=)fcgN weakly converging to x. Now let 
us split our further analysis into two distinct cases. 


Case 1. Suppose that the sequence (anfc)fceN defined in Method [T] does not converge to 0. Hence 
there exist a subsequence (without relabelling) of {an^.)k£n and a > 0 such that 


Onfc > a. (19) 

Since (x^)^^^ is bounded and ||x^ — x^+^H —)• 0 as claimed above, we get from Assumption I A2I that 

lim ||V/(x"'=) - V/(x'^'=+^)|| = 0. (20) 

k^oo 


Since = J(x"'=, ), it follows from ([5]) and (fTOl) that 


x^^ -a„,V/(x"'=) -x^'“+i 


a< 


£ dg{x'^'^^^), 


nk 


which implies in turn the expression 

+ V/(x"'=+i) - V/(x"'=) £ V/(x"'=+i) + 5ff(x"'=+i) C d{f + 5 )(x"'=+i). 




a- 


( 21 ) 


nk 


Note also that the subsequence (x”'''''^)^^^ converges weakly to x due to the fact that ||x”'' — 
3 .>T-i;+i|| 0 as A: — 00 . By passing A: —00 in ([ 2 T]), we get from ([T9]), (f 20 ]l . and Fact 12.21 that 


9 







0 G d{f + g){x), which means x G S'*. Furthermore, since the sequence ((/ + g){x^))k&i is 
decreasing due to Proposition I4.1f iii , (fTHI) is a consequence of (fTHli and (fT9]l . 

Oi. 

Case 2. Suppose now \\m.k^ooOCnk — 0- Dehne := —^ > 0 and := J (x"’*', ). 

Due to Lemma 12.41 we have 




I ^ 
rikJll S 

^rik 


lx"''' - J(x"'',q;- 




= — llx"'' — x"''^^| 


which combines with the boundedness of (x"'')fcGN to show that the sequence (x"'')^^^ is also 
bounded. It follows from the definition of Linesearch [T] that 


dn, ||V/(x"'') - V/(x"'')|| >(I||x"'' -x"''|| . (22) 

Since otnf. i 0 and both (x"'')^^!^ and (x"'')fcgp} are bounded, (|22]) together with Assumption IA2I 
tells us that limfc_).oo ||x"'' — x"''|| = 0 and thus (x"'')fcGN also weakly converges to x. Thanks to 
Assumption I A2l again, we have 

lim ||V/(x"'') - V/(x"'')|| = 0. (23) 

F*_Ilk/ II 


This and (f2^ imply that 


k^oo Q!: 


(24) 


nk 


Using ([5]) with z = x"'' — dn^V/(x"'') gives us that 
x"'' — dn^V/(x"'') — x"'' 


a. 


+ V/(x"'') G dff(x"'') + V/(x"'') C d{f + 5 )(x"''). 


n-k 


By letting k oo, we get from the latter, (l23]l . and Fact 12.21 that 0 G d{f + g){x), which 

means x G S'*. It remains to verify (I16p in this case. Indeed, we get from Lemma 12.41 that 


rv 

lx"'' -x"''|| = llx"'' - J(x"^^)|| > llx"'' - J(x"^a„J 

6 


= llx"'' — x"''"’’^ I 


This together with 


yields 




0 as A; —)• oo. Since (/ + g){x^) is decreasing due to 


Proposition 14.ll iii. we derive from the latter and (lISp that 


11 

0= lim - > lim (/+ 5 f)(x"'') - (/+ 5 f)(x*) = lim (/+ 5 f)(x^) - (/+ ff)(x*) > 0, 

K—>-00 k—^QO K—>-00 

which clearly ensures (jI6p . 

From both cases above, we have (USD and the fact that any weak accumulation point of (x^)fcgi^ 
is an element of 5*. Thanks to Fact l2.5r iiL the sequence (x^)fcGN weakly converges to some point 
in S'*. This verifies (i) of the theorem. 

To justify (ii), suppose that S'* = 0. Observe from the proof of (i) (without regarding (fTSp . (flTp . 
and (USD) that if (x*')fcgpj has any weak accumulation point then this point is an optimal solution 
as illustrated in both cases there. Since S'* = 0, any subsequence of {x^)k£f^ is unbounded and thus 
||x^|| —)• +00 as A: —)■ oo. Furthermore, note that s := + g){x^) > inix^uif g){x), where 

s exists due to fact that {{f+g){x^))kGn is decreasing by Proposition 14. 1 l ii) . If s > infa;g'^(/+gr)(x) 
then the following auxiliary set 


Siev(3:°) := |x G domfic {f + g){x) < {f + g){x^), \/k G n| 


10 









is nonempty. By applying Proposition 14.10 ') at any x G Siev(3;*^), similarly to m we also have 
{x^)ken is Fejer convergent to S'iev(a;'^)- It follows from Fact I2.5l il that the sequence is 

bounded, which is a contradiction. Hence we have s = + g){x) and complete the proof of 

the theorem. □ 

As discussed before Method [H our method improves the scheme in [33] for the particular case 
that the two maximal monotone operators considered there are V/ and dg by relaxing completely 
an additional step. Our Theorem 14.21 also loosens some unnatural assumptions imposed in [341 
Theorem 3.4(b)]. Furthermore, we obtain new information on the convergence of the cost values 
at generated sequences in this result. 


4.1 Complexity analysis of Method [T] 

In this subsection we present complexity analysis of the iterates in Method [T] When the stepsizes 
generated by Linesearch [1] are bounded below by a positive number, our analysis shows that 
the expected error from the cost value at the k-th iteration to the optimal value is 0{k~^) in 
Hilbert spaces and o{k~^) in finite dimensions, which improves the complexity of the first-order 
algorithm presented in [U Theorem 1.1]. It is worth emphasizing that the global Lipschitz continuity 
assumption on the gradient V/ used in jS] Theorem 1.1] is sufficient but not necessary for the 
boundedness from below of the stepsizes aforementioned; see our Proposition 14.41 below. Since 
afc > 0 for any k gN, this boundedness assumption actually means that liminffc^oo ctfc > 0, which 
was used before in |33| for different purposes. 

Theorem 4.3. Let {x^)keN («A;)fceN the sequences generated in Method [Tj. Suppose that 
(S'* / 0 and there exists a > 0 such that ak > a > G for all k Then we have 

(/ + <?)(x")-min(/ + 5 )(x)<^l^^^^^^i^ for all kGN. (25) 

xeH 2a k 

If in addition dim Ti < -|-oo then 

lim k[{f + g){x^) - min {f + g){x)] =0. (26) 


Proof. Pick any x* G S*, Proposition 14. 1 l il tells us that 


0 > (/ + g)ix,) - (/ + g)ix^^^) > + (1 - 2<I)llx^ - 


> 


2ac 

1 

2^' 


\x^+^-xA?-\\x^-xA?] 


for any £ G N. Since > a, we get from (1271) that 

0 > {f + g){x^) - if + g){x^"^^) > 7^(llx^+^ -x^f - \\x^ -x^f). 

2a 


Summing the above inequality over ^ = 0, l,...,fc — 1 implies that 

k-l 

f -G n'\(x^~^^') > , 

2a 


k—1 ^ 

Hf + 9){x*) - + 9){x^^^) > ^(IJa:^ - x*]]^ - JJx^ - x*]]^). 


(27) 


(28) 


e=o 
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Since (/ + g){x^) is decreasing by Proposition Id.llf iii. the latter yields 

+ 9){x^) - U + 9 ){x*)] < ^ - \\x^ - < ^\\x* - a:°|p. (29) 


Note that no matter how we choose x* G S, the optimal value (/ + g){x^) = min^jg-^ (/ + g){x) is 
fixed. Hence we get from (I29h that 


(/+s)(x )-min(/ + 9)W<2^ mt 


— x 


0||2 


1 [dist(3:°, 
2 q! k 


which verifies (I25p and completes the first part of the theorem. 

Now suppose additionally that dim'H < +oo, it follows from Theorem 14.21 that the sequence 
{x^)k£n converges strongly to some x* G S'*, he., \\x^ — x*|| — )• 0 as A: — )• oo. Take any e > 0, 
we find K £ N such that ||x^ — x*|| < e for k > K. For any i > K we get from the fact 
\\x^ — — x*|| and (l27|) that 

0 > {f + g){x^) - {f + g){x^^^) > “ ®*ll + 11^^ “ ^*11) • {\\x^~^^ - a^*|| - \\x^ - X*||) 

> —\\x^ - x*||(||x^’''^ - a;*|| - \\x^ - x*||) (30) 

ai 

> —(||x^+^ - x*|| - \\x^ - x*!!). 
a 

Now adding the above inequality over i = K,K + l^...,K + k — 1 gives us that 

K-\-k—\ 2 

k{f + 9 ){x^) - ^ U + g){x^^^) > -{\\x^^^ - a;*|| - \\x^ - x*||) > --\\x^ - x*|| > - —. 

i=K ^ a a 


Due to the decreasing property of (/ + g){x^) in Proposition Id.lf iib we get from the latter that 

k[{f + 9){x^) - U + 9){x^""^)\ > - —• 

It follows that 


limsup k[[f + g){x^) - (f + g)ix^)] = limsup(ii: + k)[{f + g){x^^'") - (f + g)ix^)] 

fc—>-oo fc—>-oo 

K + k 

< limsup — - -• — = —. 

/c—)-oo k Oi CX 

Since this inequality holds for any e > 0, we have 

0 < liminf k[{f + g){x^) - {f + g){x^)] < limsup k[{f + g){x^) - {f + g){x^)] < 0, 

k^oo k^oo 

thanks to the fact that x* G 5*. Hence we obtain limfc_,.oo k[{f + g){x^) — (/ + g){x^)] = 0, which 
verifies (j26p and completes the proof of theorem. □ 


It is worth mentioning that the rate o{k~^) was obtained |21fl23] earlier when using the proximal 
point method to solve problem ([T]) with / = 0. 0 Our result above could be considered an extension 
of some results in these papers, in particular, |22p Corollary 3.1] to the more general framework 
of (HJ with linesearch. When the the stepsizes are not bounded below by a positive constant, we 
discuss the possible validity of the same complexity as follows. 

^This important observation is pointed out from by one of the referees 
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Remark 4.1. The main question arising from the above theorem is that: Can we have the com¬ 
plexity o{k~^) of the difference (/ + g){x^) — ToSux^uif + when liminffc_).oo oik = 0? Suppose 
that (strongly) converges to some x* € S'* in finite dimension; see our Theorem 14.21 By 

analyzing carefully the proof of ([25]) in Theorem 14.31 we observe that complexity o{k~^) remains 
when the following condition holds; there exists A G [—1,1) such that 

11 ^ II1+A 

limsup- - - < + 00 , (31) 

k^oo 


which may allow Ok to approach 0. Indeed, suppose that (|3T]l is satished with some A G [—1,1), 
we find C > 0 and K ^ N such that jjx^ — < Cak for all k > K. For any e G (0,1), there 

exists Ki > K such that jjx^ ~ 3:*|| < e for all k > Ki. Moreover, it is easy to prove the existence 
of some constant H > 1 so that 


(p2 - 1) < - 1) for all p > 1. 


(32) 


Note again that 


I h 
tJO ^ 




> 1 due to the Fejer property of {x^)k£n in Theorem I4.2f ii. This 


together with (l32]) tells us that 


x'' - < 2D\\x'^ - X 


k+l *\\2 


.*||1+An. fc_^*||l-A 


k+l ^*||1-A 


— UC — X 


)• 


Hence for any ^ > Ki we get from (I27h that 


11 £ II \-\-X 

0 > (/ + p)(x*) - (/ + p)(x^+^) > D— - — -- \\x^ - 

ap 

> CD[\\x^~^ - - jjx^ - x*||^“'^). 

By adding the above inequality over = iFi, ii'i + 1,..., iFi -|- /c — 1, we have 

Ki+k-l 

kif + g){x,)- Y, {f + g)ix^^^)>CD{\\x’^+^^-x4^-^-\\x^^-x4^Y- 
e=Ki 

Due to the decreasing property of (/ -|- g){x^) in Proposition I4.11 ii). the latter implies that 
k[U + g){x,) - [f + g){x^^^^)\ > -CD\\x^^ - x,\\^-^ > -CDe^-\ 

Thus we derive the following expressions 

limsup k[{f + g){x'") - (/ + 5')(x*)] = limsup(iiri k)[{f + g){x^^^'") - (f + g)ix^)] 

fc^OO fc^OO 

< limsup = CDe^-^. 

k^oo ^ 

Since this inequality holds for any e > 0, we have limsup;._^(^ k[{f + g){x^) — (/ + p)(x*)] < 0, 
which also verifies (f26l) due to the fact that x* G S'*. □ 


It is clear that (I3ip holds when ak is bounded below by a positive number. The following simple 
example shows the possible validity of (l3T]l even when —)• 0 as A: —)• oo. Thus the complexity 
o{k~^) of the function values remains true in the example below. However, in general, checking 
()31h may be not trivial, since x* is unknown. 
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Example 4.1. Let 

f{x) ■■= 


l+p 


with 0 < p < 1 and g{x) = 5[o,oo)(3^)- 


Then a unique solution for problem ([T|) is x* = 0. Note further that for any x > 0, we have 

J{x,a) = P[o^+oo)ix — ax^) = max{x — ax^,0}. (33) 

To distinguish the iteration from the exponent in this example, we write {xk)k£n instead of {x^)k£n- 
To avoid the trivial case, suppose that Xk > 0 for all A: £ N, then we have 

0 < Xk+l = Xk- OkiXkT < Xk- 

It follows from Linesearch 1 that 

Ofckfc+i - aJfcl < 5\xk+i - Xk\. 

By mean value theorem, there exists rj £ [0,1] such that 

\Xk+i ~ = \^k+i - Xk\ ■p\riXk+i + (1 - r])xkY’~^ > \xk+i - Xk\ ■ p\xkf~^■ 


(34) 


This together with (l3l]l gives us ^ 


0 as A: 


oo, since Xk —> 0. Therefore, we may 

suppose without loss of generality that < <7 for all k. Define ak '■= — and Xk+i = J{xk, dfc), it 

u 

follows from the Linesearch 1 that 


- xl\ > S\xk+i - Xk\. 

Note that 0 < Xk+i < Xk by (l33l) and that 

0 < xf - xl_^^ < - xl~^Xk+i = xl~^{xk - Xk+l). 


(35) 


Combining this with 
that 


gives us that ^^\xk - Xk+i\ > S\xk+i 


l^T _ 1 ^ ^ 1 

* ■ ^ 7 - This is exactly (f3T]i with A = —p £ [—1,1). 
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Xk\ > 0, which implies 

□ 


Another natural question from Theorem 14.31 is that in which class of functions the stepsizes ak 
are bounded below by a positive number. Next we show that this condition is satisfied under some 
mild Lipschitz continuity assumption of V/. The first part of this result is not much surprising 
due to the similar achievement in |34l Theorem 3.4(a)]. However, the second part is a significant 
improvement when we replace the global Lipschitz continuity by the local one in finite dimensions. 


Proposition 4.4. Let {ak)k£f^ be the sequence generated by Linesearch [T] on Method [TJ The 
following statements hold: 


(i) If the gradient of f is globally Lipschitz continuous on domg' with constant L > 0, then ak > 

min {o', x} A: £ N. 

(ii) Suppose that dim'AA < +oo and S'* ^ 0. If Vf is locally Lipschitz continuous at any a; £ S* 
then there exists x* £ S* such that 


lim inf ak > min 
k^oo 



where C > 0 is a Lipschitz constant ofVf around x*. Consequently, there exists a > 0 such that 
Oik P Oi for all A: £ N. 
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Proof. To justify (i), suppose that V/ is globally Lipschitz continuous with constant L > 0. If 
ak < cr, define := — > 0 and := J{x^, dfc). It follows from the definition of Linesearch [1] 
that 


ak 


Vf{x^)-Vf{x^) 


> 6 


x^-x^ 


(36) 


which yields ||a;^ — x^|| ^ 0 for all k £ N. Moreover, due to Lipschitz assumption on V/, we get 
||V/(x^) — V/(x^)|| < L\\x^ — x^W for all A; G N. Combining the latter inequality with (f36l) gives us 
that OkL > 6, i.e., ak > ^ when ak < cr. This clearly verifies (i). 

To justify the second part, we suppose that dim^ < +oo, that S'* / 0, and that / is locally 
Lipschitz continuous at any point in 5*. By Theorem I4.2[ converges (strongly) to some 

X* G 5*. Due to the local Lipschitz continuity of V/ at x*, there exist e, £ > 0 such that 


l|V/(x) - V/(y)|| < £||x - 2/11 for all x,y £Me{x^), (37) 


where ^(.(x*) is the closed ball in T-L with center x* and radius e. Since {x^)k(m is converging 
(strongly) to x*, we find some K gN satisfying that 

Bf 

\\x^ — 2;*II < -x ^ ^ k > K (38) 

2 + 0 

with 0 G (0,1) defined in Linesearch [TJ Take any k > K, ak < cr, similarly to the first part we 
define ak ■= ^ > 0 and x^ := J{x^,ak). Thus we also have (l36l) . It follows from Lemma ITTI that 


||x*^ — x*^|| = ||x*^ — J(x*^,dfc)|| < — ||x*^ — J(x*^, ak)\\ = ^||x*^ — X 

Uk 0 

which together with (1381) implies the following expression 

II II ^ II i'll II i- II ^ 1 II r II II i- II ^ I 206^ 08 

\\ _ rp \\ _ 'T*^ _L W <F _ _ 1 _ _ rp _ , _ _ 1 _ _ 

II^L ‘£*11 _ ||‘£ II ~ ll*^ *£*11 _ /oll*^ II ~ ll*^ *£*11 _ ' p. ^ 

0 02 + 02+0 


1 , 


fc+ll 


= e. 


Hence we have x^ G B£(x*) and derive from (1371) and (f38l) that ||V/(x^) — V/(x*^)|| < £||x^ — x^||. 
Combining this with (1361) gives us that Cdk > S, i.e., ak > It follows that ak > min {o', for 


all k> K. 

Finally, since > 0 for /c G N, we obtain that ak > a := min {ai,..., ax, o'} >0 and ensure 
the last part of the proposition. The proof is complete. □ 


It is worth recalling that the assumption of Proposition I4.4f i) that V/ is globally Lipschitz 
continuous on dom^r is also snfficient for Assnmption A2. Assumptions of Proposition I4.4l iil 
are certainly not enough to guarantee Assumption A2. However, there are many broad classes 
of functions satisfying all of them. For instance, when dim'H < +oo and dom// is closed, a 
function /, which is differentiable with locally Lipschitz continuous gradient on dorng satisfies all 
the requirements; see also Proposition 12.31 

Theorem 14.31 together with Proposition 14.41 and Theorems 14.61 leads us to the following result. 
Unlike [5l Theorem 1.1], we obtain better complexity o{k~^) with linesearches in finite dimensions 
for a broader class of functions. 

Corollary 4.5. Let (x^)fcgN be the sequence generated by Method [Tj Suppose that 5* / 0. 

(i) If the gradient of f is globally Lipschitz continuous on dom//, then we have 

(/ + g){x'") - min (/ + g){x) = C>(/c"^). 
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(ii) If dimH < +00 and the gradient of f is locally Lipschitz continuous on S'*, then we have 


if + 9)ix^)+ 9)ix) = o{k ^). 


We obtain linear convergence when the stepsizes are bounded below by a positive number and 
either / or ^ is strongly convex. Recall that h : "H —)■ IR is strongly convex with constant /x > 0 if, 

h{x) > h{y) + {v, X — y) + ^\\x — y\\‘^ for all x € H, {y,v) € Gphdh. 


Theorem 4.6. Let {x^)k^^ and (afc)fcgp} be the sequences generated in Method [TJ Suppose that 
S'* / 0, that there exists a > 0 satisfying > a > 0 for all fc € N, and that either f or g is 
strongly convex with constant p, > 0. Then S* = {x*} is singleton and 


^k+l 


- xJ\ < 


+ Qf/X 


\x — X* < 




fc+1 


•v^l + o/x^ 


X — X* 


VA: e N, 


(39) 


i.e., the sequence (x^)fcgN converges (strongly) to x* with the linear rate < 1 . 

yl + 

Consequently, if either f or g is strongly convex, V f is locally Lipschitz continuous on 5*, and 
dim'R < + 00 , then {x^)keN converges linearly to the unique optimal solution. 


Proof. Since either / or g is strongly convex with constant /x > 0, / + g is also strongly convex 
with constant /x > 0. R follows that 5* is singleton {i.e., S'* = {x*}). Moreover, using Proposition 
IHKi) with X = X* and the strong convexity of / + <7 gives us that 


|x^ - x*|p >||x^+^ - x*||^ + 2 afc[(/ + 5 )(x^+^) - if + g){x^)] 

>||x^+^ — x*|p + akp.\\x’^~^^ — x*|p > (1 + a/x)||x^+^ — x*|p. 


R follows that 


\rf.k+l O' II < 

vO tXy 3)t ^ 


y/\ + O/i 


X — X* < 


\ k-\-l 


y y'l + o/x J 


X — X* 


which verihes (j39[) and thus completes the proof of the theorem. 


□ 


Since the condition x = J{x,a) for ce > 0 is necessary and sufficient for x to be an optimal solution 
to problem ([T|), it is interesting to study the complexity of ||x^ — J{x^,ak)\\ in our Method [TJ The 
velocity of the convergence obtained below is not affected by the behavior of the stepsizes a^. 

Theorem 4.7. Let {x^)k£n and {ak)k€N be the sequences generated from Method [TJ Then we 
have 

liminf • ||x^ — J(x^, afc)|| = 0. (40) 

fc^OO 

Proof. R (j40p does not hold, then we may find a number e > 0 such that for some fixed RT G N 

large enough, we have ||x^ — J{x^,ak)\\ > —p for all k> K. Thus, 

V A: 

00 00 ^ 

^ ||x*^ - J(x^,afc)f > ^ - =+ 00 . (41) 

k=K k=K ^ 
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On the other hand, using (fT0|) and Proposition 14. llf iii. we get, for all k > K, 

||x" - J(x^ ak)f = ||x" - < ^ [(/ + - (/ + 5)(x^+i) 

a 


< 


1-5 


1-5 1 

[f + g){x^) - [f + g){x’^+^) 


where we have used in the last inequality that < cr for all A; £ N, which follows from Linesearch 
[TJ Hence, we have 


- J{x'",ak)f < [if + 9){x^) - if + 9){x*)] < + 00 , 

k=K 

which contradicts (I4ip . The proof is complete. □ 


4.2 A fast multistep forward-backward method with Linesearch [T] 

In the spirit of the classical work of Nesterov m many accelerated multistep versions have been 
proposed in the literature for the forward-backward iteration, but to the best of our knowledge all 
of them have to employ the global Lipschitz continuity assumption on V/; see, e.g., msm- In this 
subsection, by following these ideas and assuming no Lipschitz continuity on V/, we present a fast 
version of the proximal forward-backward method with Linesearch [U improving the convergence 
result of Theorem l4.3l for Method [H In [Hl5ll29j this kind of fast versions usually demands Lipschitz 
assumption over V/ to establish convergence of this method. Here we modify the method by adding 
a linesearch and an extra projection step in (j43p below to avoid the requirements aforementioned. 
For simplicity, we suppose H := domg' is closed in this section. 


Method 2. 

Initialization Step. Take x~^ = £ dom^r, to = 1, 6 £ (0,1), a_i = a and 5 £ (0,1/2). 

Iterative Step. Given and x^, set 


i + yiTd^ 

4+1 - 2 

/ = x’^ + {x'^ - f = Pn{y’^) 

= Jiy^,ak) ■■=prox^^g{y^ - akVfiy’")) 

with Ok := Linesearch [T])i/^, afc_i, 0,5). 

Stop Criteria. If x^~^^ = y^, then stop. 


(42) 

(43) 

(44) 


Note that from (f43P and (f44|) . y^ and x^ belong to domgi for all A: £ N and as a direct consequence 
of Lemma EH ak satisfying (1451) is always positive and nonincreasing. Moreover, it is similar to 
Method [T] that if = y^ then is an optimal solution. An important inequality for our 
further study from Linesearch [1] is 




V/(x^+i)-V/(y") 


< <5 


x^+^ - y’' 


(45) 


with 5 £ (0,1/2). We also need some auxiliary results before establishing the convergence results. 


17 























Lemma 4.8. The positive sequence (tfc)fceN generated by Method [2] via (|42ll satisfies, for all 
A; e N, 


(ii) ti+i -4+1 = ti- 

Proof. The proof easily follows by induction argument. □ 

Proposition 4.9. Let ak be defined in Method [2] and x G dom^f. Then we have 

if + 9 ){x) - if + - xf - \\y^ - for all keN. (46) 

k~\~ 1 

Proof. First note from (j5|) with z = f{y^) that - - - -V/(y^) G dg{x^^^). Then, 

ak 

g{x) - g{x^"^^) > - Vf{y^),x - (47) 

for all X G dom^r. The convexity of / implies that 


f{x) — f{y) > {Vf{y),x — y) for all x G dom/ and y G dom^r. (48) 

By summing (I47|) and (I48p with y = y^ £ Q = dom g, we obtain that 

/ ^k k I 1 \ 

(/ + g){x) >f{f) + 5(^"+') + - ^f{f),x - x^+^j + {Vfif),x - f) 

=/(y") + 5(x"+') + — (f- x>^+\x - + {Vf{f),x’^^^ - f) 

=f{f) + g{x'^^^) + — (f- x^+\x - + {Vf{f) - V/(x^+^), - f) 

ak \ / 

+ {Vf{x^+^),x^+^ -f) 

>f{f) + gix'^^^) + — X - x^+i\ - —||x^+^ - ff 

ak \ ! ak 

+ (V/(x'=+'),x'^+i-y'=), 


where the last inequality follows from (|45p . Rearranging the inequality gives us that 

Ifit - - x) >ak\f{ft) + g{x^^^) - (/ + g){x)] - (5||x^+^ - y’^f 

+ ak{Vf{x^+^),x^+^-f). 


(49) 


Observe that 2{y^ — — x) = \\y^ — x|p — — x|p — \\y^ — By combining the 

above equality with dM]), we have 


f - x|p - - x|p >2ak 


f{y^) + g{x^+^) - (/ + g){x) + (V/(x^+i), 
+ (l-2,5)||y'^-x'^+if 


>‘^ak 


/(r)+5(x"+')-(/ + y)(x) + (V/(x''+^),x"+^-r) . (50) 


fc+lN ™Al+l ;r,k\ 
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It follows from (HSil with x = and y = that f [y^) — f {x^~^^) > {Vf{x^~^^),y^ — which 

together with ([501) implies 


- xlp - - xlp >2ak 


— ‘2oik 


f{f) + g{x^+^) - (/ + g){x) + /(x"+i) - fit) 
if + 9)ix^^^) - if + g)ix) 


Since ||y^ — x|| < ||?/^ — x|| for all x G domgr due to (l43]l . we get from the latter (|46|) and complete 
the proof of the proposition. □ 

In the next result we establish a better complexity for Method [2] than Method [1] in Theorem 
oi under a similar assumption. 

Theorem 4.10. Let (x^)fcgisj and iak)k£N be the sequences generated in Method [2J Suppose that 
S'* / 0 and there is a > 0 such that > a > 0 for all A: G N. Then we have 


if+g)ix )-iamif+g)ix) < 

x&n 


2 

a 


— a:*|P + 2(7 


if + 9)ix ) - min(/ + g)ix) 


(fc+l)2 


for all A; G N. 


Proof. To justify, pick any x* G S*. By Lemma iTHI ii and the convexity of g, we have tk+i > 1 and 
thus X := f^tx^ + (1 “ tti)x^ G domg. Applying Proposition 14.91 for this x gives us that 

^ ^ II" 11^^" +(1" ^fc+i) II) 

< (/ + 9)itti^* + (1 - ^fc+i) - (/ + 9)ix^^^) 

< Afc+i(/ + 9)ix*) + (1 - Afc+i)(/ + 9)ix’') - (/ + 9)ix’"^^). 

After rearrangement, we obtain 

(1-Afc+i) if + 9)ix’") - if + g)ix:,) - if + g)ix^^^) - if + g)ix^) 


> 


2akt 


k+l 


tk+ix^^^ - (x* + (4+1 - I)x'‘) - Afc+i/ - (x* + (ifc+1 - l)x'') 


By multiplying by to the above inequality and using ()43l) and Lemma l4.8l fiii. we have 
^ (^\\tk+ix’"^^ - (x* + ( 4+1 - l)x^)f - ||Afc+i/ - (x* + ( 4+1 - l)x'')f ^ 
|4+ix^+^ - ( 4+1 - l)x^ - x*|p - ||4x^ - (4 - I)x^“^ - x*|p^ 


2ak 

< (a1+i - 4+i) [(/ + 9)ix'') - (/ + 9)ix*) 


— t 


k+l 


if + 9)ix^^^) - if + 9)ix*) 


- y-2 


if + 9)ix^) - if + g)ix*) 


-1 


fc+1 


if + 9)ix’"^^) - if + 9)ix*) 


It follows that 

\\tkx'" - itk - l)x^“^ - x*|p - ||4+ix*'+^ - ( 4+1 - Ijx*' - x*|p 
>2ak(tl^i if + 9)ix’'^^)-if + g)ix*)^ 

> 2ak+itl+i \if + g)ix'^^^) - (/ + 5)(a;*)l - ‘^Oiktl [(/ + g)ix'^) - if + 9)ix^) 


-tl if + 9)ix'') - if + 9)ix*) 
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where the last inequality follows from the facts that Uk > ctk+i = Linesearch l{y^,ak,0,d) and 
(/ + “ (/ + g){x*) > 0. Reordering the above inequality and applying it inductively yield 

2afc+iti+i {f + - {f + g){x^) 

< ||4+ix^+^ - (4+1 - + 2ak+itl+i if + g){x^"^^) - (/ + g)ix^) 

< ||4x^ - (4 - + 2aktl (/ + g)ix^) - (/ + g)ix^) 

< ... < \\tox° - {to - l)x~^ - + 200^0 [(/ + g){x^) - if + 5)(a;*)] 

= ||x° - x*||^ + 2ao [(/ + g){x^) - if + g)ix*)] , 

which readily imply 2aktl[if + g)ix^) - if + g)ix*)] < ||x°-x*P + 2(T [{f + g)ix^) - if + g)ix*)] 
Using this inequality together with Lemma l4.8f ii gives us that 


if + g)ix ) - min (/ + g){x) < 


1 


<- 


2akti 
2 
a 


+ 2cj 


if + g)ix°) - min (f + g)(x) 

x^T-i 


|x° — X*|P + 2(7 


if + g)ix ) - min(/ + g){x) 

xGH 


ik + iy 

for all X* € S'* and thus verifies (j46p . The proof of the theorem is complete. 


□ 


This theorem shows that the expected error of the iterates generated by Method [2] after k 
iterations is 0{k~‘^) when the stepsizes are bounded below by a positive constant. Similarly to 
Proposition 14.4( we prove in the next result that such a requirement is satisfied under global 
Lipschitz assumption on the gradient of /. The complexity o{k~‘^) for the accelerated scheme 
similarly to (j42p ~(45) has been obtained recently in Odl] under the global Lipschitz assumption. 
It would be interesting to combine their techniques with ours to derive similar complexity under 
the weaker assumption of local Lipschitz continuity as in Proposition I4.4f iil . 

Proposition 4.11. Let (afc)fcGN be the sequence generated by Linesearch [T] on Method [2l If 
the gradient of f is globally Lipschitz continuous on domg' then there exists some (U > 0 such that 
Oik> ot for all /c G N. 


Proof. Suppose that V/ is globally Lipschitz continuous on domgi with constant L > 0. Since ak is 
nonnegative and decreasing, limfc_j.oo oik = a exists. If a < we may find R G N such that Ok < ^ 

for all A: > K. Define further dfc := -^ > 0, and := J{y^,ak) = prox^^g(y^—dfcV/(y^)) G domy. 
If ak < Ofc-i for k > K, it follows from the definition of Linesearch [T]that 


ak 


^fif)-'^fif 


> 6 


y -y 


(51) 


Due to the fact V/ is Lipschitz continuous on domy with constant L, we get from (EH) that 
akL\\y^ — y^ll > (5||y^ — y^\\. Thus at > which is a contradiction. Hence at > ak-i, i.e., 
ak = ak-i for all k > K. This tells us that ax = a > 0 whenever a < ^. Thus we always have 
q; > 0 and complete the proof. □ 


Let us complete the section with a direct consequence of the above proposition and Theorem 14. lUI 
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Corollary 4.12. Let be the sequence generated by Method [2]. Suppose that ^ 9 and 

the gradient of f is Lipschitz continuous on domg'. Then we have 

(/ + 9){x’') - min(/ + g){x) = 0{{k + 1 )"^). 

5 The forward-backward method with Linesearch [2] 

Method [T]requires to evaluate the resolvent of dg inside Linesearch [T] at each step of the iteration. 
When the proximal step is not easy to compute, Method [T]may be inefficient. To overcome this 
drawback, we propose here a modification of the forward-backward method by using Linesearch!^ 
which involves only one computation of the resolvent of dg for all steps of this linesearch. We also 
prove that the sequence generated by this method is weakly convergent to a solution of problem 
©• 


Method 3. 


Initialization Step. Take x^ G dom 51 and 6 G (0,1). 

Iterative Step. Set 


Jk = proXg(x*^ -Vfix^)) 

(52) 


(53) 

with fdk := Linesearch [2lx^ .6). 

Stop Criteria. If x^~^^ = x^, then stop. 



Thanks to Lemma 13.21 and the convexity of g, we note that G dom^ inductively. Moreover, it 
follows from Linesearch [2] that 

if + < if + g){x’") - Pk g{x^)-g{Jk) -/3fc(V/(a:''),x^ - J^) + y ||x''- . (54) 

Next we obtain some similar results for Method [3] to the ones in Section 3 for Method [TJ The 
following proposition is corresponding to Proposition 14.11 


Proposition 5.1. Let x € dom^. Then we have 
Wx’^+^-xf < \\x’^-xf+2 \{f + g)ix^) - (/ + 5 )(x^+i) 


+2/5fc 


if+ 9)ix) - if+ 9)ix^) 


yk G N. 


Proof. Fix any x G dom g and set Ak := x^|p + ||a;^—x|p — x|p = 2{x^—x^~^^,x^—x). 

Moreover, we get from ()53p that 


Ak 

2^k 


= {x^ - Jk,x^ -x) = {Vfix^),x^ - x) + {x^ - Jk - Vfix^),x'^ - x) 

= {Vfix'^),x^ -x) + {x^ -Jk- Vfix^), Jk-x) + ix’^ -Jk- Vfix^),x’^ - Jk) 

= {Vf{x^),x^ -x) + {x^ -Jk- Vf{x^), Jk-x)- {Vfix^),x^ - Jk) + \\x^ - Jfef. 
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Observe from (|^ that — V/(x^) — Jk £ dg{Jk)- By applying (jl|) and (1^ to the above 
expression, we have 

Ah. ^ _ f j- r,{ T,^ - _ /V7 f 

2Pk 


> fix^) - fix) + giJk) - g{x) - {Vfix^),x^ - 4) + \\x^ - 4^ 

> fixA +9iJk) - if + 9)ix) + -^\if + 9)ix^^A - if +9)ix'")] +9ixA - 9iJk) + \\\x'^ - Jkf 

Pk^ J I 


1 

^Jk 


1 , 


if + 9)ix^^A - if+9)ixA\ + - Jkf 


= if + 9)ixA - if + 9)ix) 

It follows that 

\\x^^^ - xf - x’^f - Pj,\\x^ - Jkf + 2 [(/ + 5)(xk - (/ + 

+ 2Pk if + 9)ix) - if + g)ixA ■ 

Since x^^^ — x^ = PkiJk — x^) by ([55|1 and P‘1 < Pk, we conclude that 

lk"+' - ^Ik"' - + iAk - h)\\x^ -Jk\? + 2\if + g)ix^) -if + g)ix^^^) 


+ 2Pk 


if + 9)ix) - if + g)ix^) 


<||^fc - xk + 2 (/ + g)ix^) -if + g)ix'^^^) 


+ 2Pk 


(/ + 9)W-(/+9)M 


as desired. The proof is complete. 


□ 


It is worth noting that using Proposition 15.11 with x = G dom^f gives us that 

if + 5)(xk - (/ + > ^Ik"+' - ^"k > 0, (55) 

which shows that Methodic is also a descent method. 

Next we establish the main result of this section whose statement is similar to Theorem 14.21 

Theorem 5.2. Let (x^)fcgN be the sequence generated by Method [3l The following statements 
hold: 

(i) If lb then (x^)fcGN is quasi-Fejer convergent to S'* and weakly converges to a point in S*. 

(ii) If S* = 0 then we have 

lim llx^'ll = Too and Ihn if + g)ix^) = inf (/+ 5 r)(x). (56) 

k^oQ k^oQ xG'H 


Proof. To justify (i), suppose that S* 7 ^ 0. By employing Proposition 15.11 at x = x* G S* C dom^, 
we have 


- x*k < Ik^ - x*A + 2 if + g)ixA - if + 


for all A: G N. (57) 
It follows from (IS5]) that := 2 [(/ + g)ix^) — if + fl')(a^^^^)] > 0. Moreover, observe that 

00 00 

Y^ek=2Y,\if + 9)ixA -if + 5)(^"+kl < 2 [(/ + g)ix^) - l^Jf + <7)(^"+k' 


fc =0 k=0 

<2 


[if + 9)ix°) - if + g)ix*) 


< + 00 . 
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This together with (EZD tells us that the sequence is quasi-FejCT convergent to 5* via 

Definition 12. 11 By Fact l2.5r i). this sequence is bounded and hence it has weak accumulation points. 
Let X be a weak accumulation point of {x^)keN- Hence there exists a subsequence {x‘^^)keN of 
converging weakly to x. Now we distinguish our analysis into two cases. 


Case 1. The sequence does not converge to 0, i.e., there exist some (3 > 0 and a 

subsequence of {l3nk)k£'N (without relabelling) such that 

Pn,>(3, VfceN. (58) 

By using Proposition 15.11 with x = x* G 5*, we get 


Pk 


{f + g){x^) - (/ + ff)(x*) <-(||x'= -x*f - ||x''+^ - x*f) + {f + g){x'^) - {f + g){x’‘^^). 


Summing from /c = 0 to m in the above inequality implies 


if + g){x'") - if + g)ix*) 


k=0 


<^(ll^° - ) + (/ + g)ix^) - (/ + 5)(^™+') 


<-||x° - X*f + (/ + g)ix^) -if + g)ix^)- 


By taking m —)• oo and using the fact that (/ + g)ix^) > (/ + 5 ')(x*), we obtain that 


^/3nfc [(/ + 5')(a:"'') - if + g)ix*)] < if + g)ix'") - if + g)ix*) 


< + 00 , 


fc =0 


k=0 


which together with (l58l) establishes that lim (/ + 5 )(x”'=) = (/ + 5 )(x*). Since f + g is lower 

k^oo 

semicontinuous on domgf, it is also weakly l.s.c. due to the convexity oi f + g- It follows from the 
last equality that 

if + g)ix*) < if + g)ix) < liminf(/ + 5r)(x”'') = lim if + g)ix'^^) = if + g)ix*), 

k^oo K—>-oo 


which yields (/ + g)ix) = (/ + 5 ')(x*) and thus x G S'*. 

' 0k 

Case 2. lim 0k = 0. Define /3fc := — > 0 and 

k-i-QO u 

f := x^ - hix^ - Jk) = (1 - h)x'^ + kJk. (59) 

It follows from the definition of Linesearch 2 that 

if + g)iy^) >if + g)ix^) - Hgix^) - giJk)] - 4(v/(x^), x" - Jk) + ^||x" - Jkf. (eo) 

This together with (j3|) and (l5^ gives us that 

0 > - ;Sfc(V/(x"), x" - Jk) + if + g)ix^) - (/ + g)if) - hUx'^) - giJk)] + y - Jkf 

= - 4(V/(x^),x^ - Jk) + fixf - fif) + gixf - giyf - 0k[gix^) - giJk)] + y - Jkf 

> - 0k{Vfixf,x’^ - Jk) + {Vfif),x’^ - f) + ^Wx’^ - Jkf 
+ gixf - (1 - 0k)gixf - 0kgiJk) - 0k[gixf - giJk)] 

= 0k{Vfif) - Vfix^),x^ - Jk) + ^\\x^ - Jkf. 


23 








We obtain that 

^\\x^ - Juf < hW^fiv^) - V/(x'=)|| • llx"^ - 411, 

which yields 

i||i‘-*ll<l|V/(S‘)-V/(i*)||. (61) 

Since proXg(-) is nonexpansive, we get from (l5^ that \\Jk — 411 < \\x^ 

Due to Assumption A2 and the boundedness of {x^)k£n, the latter tells us that {Jk)k€N is also 
bounded. This together with (1^^) and the fact /3k ^ 0 implies that ||y^ — x^|| —)■ 0 as k —)■ oo. 
Since V/ is uniformly continuous on bounded sets, we get ||V/(y^) — V/(x^)|| —)• 0 as A; —)• oo and 
derive from (161 p that 

lim ||x^ — Jk\\ = 0, (62) 

fc^OO 

Since V/ is uniformly continuous on bounded sets, (f62p implies 

hm ||V/(x'=)-V/(4)|| =0. (63) 

k—¥oo 

Using ([5]) with z = x^ — V/(x^) gives us that 

x^-Jk + Vf{Jk) - Vf{x^) G V/(4) + dg{Jk) C d{f + g){Jk). 

By passing to the limit over the subsequence {nk)ke'N ia the above inclusion, we get from Fact 12.21 
(l62l) . and (l63l) that 0 G d{f + g){x), which implies x G S'*- 

In all possible cases above, any weak accumulation point of {x^)k^N belongs to S*. Fact I2.5r ii) 
tells us that (x^)^^^ converges weakly to an optimal solution in S*. Thus this completes the proof of 
(i). Moreover, the proof of part (ii) is quite similar to the arguments used to prove Theorem l4.2l fiib 
We omit the detail and complete the proof. □ 

From the view of (j56l) and also our Theorem 14.21 it is natural to question that whether 

lim (/+ 5 )(x^) = min(/+ 5 r)(x) (64) 

k^oo x^ti 

in the case S* ^ 0. We do not know the answer in general, but when either f + g is continuous on 
the domg' in finite dimensions or the sequence {I3k)k&n is bounded below by a positive constant, 
the equality (j64[) is true with some further complexity discussed in the next subsection. 


5.1 Complexity analysis of Method [3] 


In this subsection we establish the complexity of Method [3] with a similar rate to Theorem 14.31 as 
follows. 

Theorem 5.3. Let {x^)ken and (/3fc)fcgN sequences generated in Method [3l Suppose that 

S'* / 0 and there is some /3 > 0 satisfying /3k > (3 > 0 for all A; G N. Then for all k G N 


if + 9){x^) - min (/ + g){x) < 


^ [dist(x°, S*)]^ + 2 


(/ + 5)(a^°) - min(/ + g){x) 
k 


If in addition dim 4 < +oo then we have 


(65) 


lim k 
fc —>-00 


(/ + g){x^) - min (/ + g){x) 
xeK 


= 0 . 


( 66 ) 
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Proof. By using Proposition 15.11 at ^ G N and x* G 5*, we get 
0 >(/+5)(a;*) - if+9){x^^^) 


>- 


1 

-m 

1 

-2/3 




if + g){x^^^) - if + g){x^)\) 


>— ( \\x^^^ -x*|p - + 2 if + g)ix^^^) - if + g)ix^) 


(67) 


for all G N. Summing the above inequality (|67p . over = 0,1,..., fc — 1, we have 
k—1 ^ 

Y1 [if + g)i^*) - if + g)ix^^^) > ^ {\\x’" -x4‘^ - \\x^ - x4‘^ + 2[if + g)ix’^) - if + g)ix^ 


£=0 


> 


2/3 


X — x* 


- \\x° - + 2 [if + g)ix^) - if + g)ix^ 


( 68 ) 


Noting that if + g()(x^+^) > (/ + g)ix^) for all = 0,..., A: — 1 by (1^ . we obtain from (f55|l that 


if + g)ix*) - if + g)ix^) 


> 




_ /y> _ 

•3^ ^ 


- x^W^ + 2 [if + g)ix^) - if + g)ia 


which clearly implies the following expression 

1 ||x° - + 2 [(/ + g)ix^) - if + 5')(a;*)] 


if + g)ix ) - if + g)ix*) < 


2/3 


k 


(69) 


for all X* G S'*. (l65]l is obtained. 

To justify ([HH]) when dim'H < + 00 , suppose that (x^)fcGN converges (strongly) to some x* G S* 
by Theorem 15.21 Hence for any e > 0 there exists some 77 > 0 such that 

||x^ —x*||<e and (/+ 5 ')(x^) — (/+ 5 ()(x*) < e for all k>K, (70) 

where the second inequality follows from the recent estimate (1691) . Adding (1671) for i = K,K + 
1,..., K + k — 1 and noting that 


K+k-l 


£=K 


[if + g)ix*) - if + g)ix^^^) > ^ (11 


> ^ - \\x^ - X*|P + 2 


if + g)ix^^’^)-if + 9)ix^) 


A'^ 


Since (/ + g)ix^^^) > (/ + g)ix^~^^) for all 7 = 77, i7 + 1,..., i7 + /c — 1 by (1551) . we get from the 
latter and (ITOj) that 

k[if + g)ix*)-if+9)ix^^^)] > ^(-||x^-x*f+ 2[(/ + ff)(x*)-(/+ 5 )(x'^)]) > ^i-s^-2s). 


It follows that 


limsup k[if + g)ix^) - if + g)ix^)] = limsup(77 + k)[if + g)ix^^^) - if + g)ix^)] 

k^oo k^oo 

K + k e^ + 2£ £^+ 2£ 

< limsup —;-•--— = --—. 

- k^J k 2/3 2/3 

Since this inequality holds for any e > 0, we have limsupfc_^oo ^[(/ + 9)ix^) ~ if + 5 )(x*)] < 0. 
Note that (/ + g)ix^) — (/ + g)ix^) > 0 for all 7 G N, we get (1661) and thus complete the proof of 
theorem. □ 
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Similarly to Lemma 14.41 we present some sufficient conditions for the below boundedness by a 
positive constant of the stepsize generated by Linesearch [2l 

Proposition 5.4. Let {f3k)keN be the sequence generated by Linesearch [2] on Method [3j The 
following statements hold: 

(i) If the gradient of f is globally Lipschitz continuous on dom g with constant L > 0, then f5k > 
min {l, for all fe G N. 

(ii) Suppose that dim'H < +oo and S'* / 0. If Vf is locally Lipschitz continuous at any x G S* 
then there exists x* G S* such that 

liminf/3fc > min|l,^|, (71) 

k^oo K lL, ) 

where £ > 0 is a Lipschitz constant ofV f around x*. Consequently, there exists ft > 0 such that 
fdk > P for all A: G N. 


Proof. First let us verify (i) by supposing that the gradient of / is globally Lipschitz continuous 

^ Pk 

on dom^r with constant L > 0. Define Pk ■= -^ > 0 and 

6 

■■= PkJk + (1 - = x^- Pkix’^ - Jk). (72) 

If /3fc < 1, we get from Linesearch 2 that 


(/ + 9){y^) > (/ + 9){x^) - h[9{x'^) - 9{Jk)] - /3fc(V/(x^),x^ - JP) + ^||x" - Jk\\\ 

which together with (|72p and that x^ — 7 ^ 0 implies that y^ 7 ^ x^. Furthermore, it is similar 

to (|6T]l in the proof of Theorem 15.21 that I\\x^ — Jp] < ||V/(y^) — V/(x^)||. Due to the Lipschitz 
continuity with constant L of V/, we get from the latter and (j72p that 

l\\x^-Jk\\<L\\x^-f\\=LPk\\x'^-Jk\\. 

Since x^ — 7 ^ 0, the inequality above yields Pk P jp and thus Pk > when Pk < 1. It follows 

that Pk > min{l, as desired. 

To verify the second part, suppose that dim77 < + 00 , S* 7 ^ 0, and that V/ is locally Lipschitz 
continuous at any x G 5*. By Theorem 15.21 suppose that {x^)k& (strongly) converges to x* G S'*. 
Hence there exist e,£ > 0 such that 


||V/(x) - V/(y)|| < £||x - y|| for all x,yeMe{xp. 

Since x^ —>■ x* as A: —)• 00 , we find K > 0 such that ||x^ — x*|| < s for all k > K. Pick any k > K, 
if /3fc < 1, dehne Pk = -^ > 0 and = PkJk + (1 — Pk)x^ ■ Similarly to the above argument of the 
first part, we have x^ — 7 ^ 0 and 


1 , 


\x^-Jk\\ < ||V/(y'^)-V/(x' 


(73) 


We consider two cases as in Theorem 15.21 as below: 
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Case 1. The sequence {l3k)ke'M is bounded below by a positive number (3 > 0. Thanks to (l53|) we 
have 


Ix’^-JkW = 


l| < II 


(3k 


(3 


13k 


as /c — )• oo. It follows that \\x^ ~ y II = “ttII®^ “ •^fcll Oj which tells us that {((^(keN is converging 

0 


to X*. Hence there exists Ki > K such that G B£(x*) for all k > Ki. By combining this with 
(ESI), we derive 


- JkW < C,\\t -x^\\= Ch\\x^ - Jfcll for all k > Ki. 


Since ||x^ — Jfc|| / 0, the latter gives us that ^ < £/3fc, i.e., /3fc > ^ for all k > Ki. 

Case 2. The sequence {/3k)k&fi is not bounded below by a positive number /3. Hence we may 
find a subsequence (no labeling) {/3k)keN converging to 0. It is similar to the proof of Case 2 in 
Theorem 15.21 that {x^)k£n and {Jk)k£N are bounded. It follows that 

lim ||x*^ — y*^|| = lim ^||x^ — Jk\\ = 0. 

k^oo k^oo u 


Thus the sequence {y^)keN is converging to x*. Repeating the corresponding part in the proof of 
Case 1 above, we also have /3fe > ^ for any large k, which is the contradiction. 

From the analysis of both cases above, we find Ki > 0 such that /3fc > ^ if /3fc < 1 for any 
k > Ki. This means /3fc > min{l, for k > Ki. The proof is complete. □ 


Let us complete the section by presenting a corresponding corollary to Corollary 14.51 which is 
easily derived from Theorem 15.31 and Proposition 15.41 

Corollary 5.5. Let {x^)kGN be the sequence generated by Method [3j Suppose that / 0. 

(i) If the gradient of f is globally Lipschitz continuous on dom^r, then 


if + g){x ) - ioam{f + g){x) = 0{k ). 

x£Ti 

(ii) If ddmli < +00 and the gradient of f is locally Lipschitz continuous on S'*, then we have 

if + 9)ix^) - min(/ + g)ix) = o(A:"^). 


6 Conclusions 

In Hilbert spaces, it is well-known that convexity on both functions and global Lipschitz conti¬ 
nuity on the gradient of / are sufficient for providing convergence of the sequence generated by 
the forward-backward splitting methods for solving problem ([T|). However, the Lipschitz assump¬ 
tion is usually a restriction in many particular circumstances. In this work we dealt with weak 
convergence of the forward-backward splitting method for convex optimization problems by taking 
the advantage of the linesearches. This not only eliminates the serious drawback of estimating 
the Lipschitz constant to choose the stepsize in ([2]) but also establishes many complexity results 
without imposing the Lipschitz assumption. Our schemes through the linesearches provide rigorous 
and implementable ways of updating the iterates, which can be easily adapted for applications. 
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We hope that this study will serve as a basis for future research on other efficient variants of the 
forward-backward splitting iteration. In particular we find possibility to develop our methods to 
the descent coordinate gradient method m for solving structured convex optimization problems. 
Moreover, we discuss in separate papers the cases when / or g are nonconvex following the ideas 
exposed in [9] and even removing the differentiability of / and adding dynamic choices of the 
stepsizes with conditional and deflected techniques combining the ideas in [GUSoIES]. We are also 
looking to the incremental (sub)gradient method like |28j for problem ([1]), when / is the sum of a 
large number of functions. An interesting project, suggested by a referee, that we are pursuing is 
to study possible complexity o{k~‘^) and the weak convergence of Method2 without assuming the 
global Lipschitz continuity on the gradient of the smooth function as in mm- 
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